56 research outputs found
Neural Machine Translation with Byte-Level Subwords
Almost all existing machine translation models are built on top of
character-based vocabularies: characters, subwords or words. Rare characters
from noisy text or character-rich languages such as Japanese and Chinese
however can unnecessarily take up vocabulary slots and limit its compactness.
Representing text at the level of bytes and using the 256 byte set as
vocabulary is a potential solution to this issue. High computational cost has
however prevented it from being widely deployed or used in practice. In this
paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE),
which is compacter than character vocabulary and has no out-of-vocabulary
tokens, but is more efficient than using pure bytes only is. We claim that
contextualizing BBPE embeddings is necessary, which can be implemented by a
convolutional or recurrent layer. Our experiments show that BBPE has comparable
performance to BPE while its size is only 1/8 of that for BPE. In the
multilingual setting, BBPE maximizes vocabulary sharing across many languages
and achieves better translation quality. Moreover, we show that BBPE enables
transferring models between languages with non-overlapping character sets
VizSeq: A Visual Analysis Toolkit for Text Generation Tasks
Automatic evaluation of text generation tasks (e.g. machine translation, text
summarization, image captioning and video description) usually relies heavily
on task-specific metrics, such as BLEU and ROUGE. They, however, are abstract
numbers and are not perfectly aligned with human assessment. This suggests
inspecting detailed examples as a complement to identify system error patterns.
In this paper, we present VizSeq, a visual analysis toolkit for instance-level
and corpus-level system evaluation on a wide variety of text generation tasks.
It supports multimodal sources and multiple text references, providing
visualization in Jupyter notebook or a web app interface. It can be used
locally or deployed onto public servers for centralized data hosting and
benchmarking. It covers most common n-gram based metrics accelerated with
multiprocessing, and also provides latest embedding-based metrics such as
BERTScore
- …